1. Field of the Invention
This invention relates in general to embedded media markers, which signify the existence of media associated with a part of a document containing the embedded media marker, and also to methods and systems for generating an article including aforesaid embedded media markers.
2. Description of the Related Art
Although paper is one of the most widely used devices for viewing information, it cannot play dynamic media such as video and audio. On the other hand, cell phones are increasingly used to play audio and video but cannot match paper's high resolution, large display size, flexibility in spatial organization, outdoor-readability and robustness for static content. It is now possible to combine the two, using image recognition technology to link paper documents to corresponding dynamic media. A cell phone camera is used to capture an image of a document patch. The document patch is identified using features in the image, and digital media linked to that location in the document is retrieved and then played on the cell phone.
A common method for creating this type of media link on a paper document is to print coded markers on the document. Examples of such markers include two-dimensional bar codes and printed patterns of dots, described in Hecht, B., M. Rohs, J. Schöning, and A. Krüger, Wikeye—Using Magic Lenses to Explore Georeferenced Wikipedia Content, Proceedings of the 3rd International Workshop on Pervasive Mobile Interaction Devices (PERMID), pp 6-10, both of which are well known to persons of skill in the art. However, these markers are visually obtrusive and interfere with the document content layout. DataGlyphs, described in Hecht D. L., Embedded Data Glyph Technology for Hardcopy Digital Documents, SPIE—Color Hard Copy and Graphics Arts III, Vol. 2171. February 1994, pp 341-352, overcome these problems by printing a nearly invisible machine-recognizable pattern on the paper. However, this type of marker requires high resolution printers and cameras to identify document locations. Electronic markers such as RFID, described, for example, in Reilly, D., M. Rodgers, R. Argue, et al., Marked-up maps: combining paper maps and electronic information resources, Personal and Ubiquitous Computing, 2006, 10(4): pp 215-226, can also be used, but result in substantial increase of production costs.
Other systems use features of the document content itself for identifying the relevant document patch and using it to create a media link. For example, HotPaper, described in Erol, B., Emilio Antunez, and J. J. Hull, HOTPAPER: multimedia interaction with paper using mobile phones, Proceedings of Multimedia '08, pp 399-408, and Mobile Retriever, described in Liu, X. and D. Doermann, Mobile Retriever: access to digital documents from their physical source, Int. J. Doc. Anal. Recognit., 2008. 11(1): pp 19-27, use features based on document text such as the spatial layout of words. Other systems such Bookmarkr, described in Henze, N. and S. Boll, Snap and share your photobooks, Proceedings of Multimedia '08, pp 409-418, and MapSnapper, described in Hare, J., P. Lewis, L. Gordon, and G. Hart, MapSnapper: Engineering an Efficient Algorithm for Matching Images of Maps from Mobile Phones, Proceedings of the SPIE, Volume 6820, pp 68200L-68200L-11 (2008), use pixel level image features, such as the SIFT algorithm, described in Lowe, D. G., Distinctive Image Features from Scale-Invariant Keypoints, Int. J. Comput. Vision, 2004, 60(2): pp 91-110, to recognize generic document content such as pictures and graphic elements. With these systems, visually obtrusive marks are not required for identification.
However, both the conventional marker-based methods and content-based methods lack media type indicators. Although bar codes and Data Glyphs are visible, they do not directly indicate the existence or type of media associated with them. When content-based feature are used, there is no on-paper indication to the user that there is media linked to the document. As a result, a user of the aforesaid HotPaper has to pan a camera phone over the paper document to look for hotspots until a feedback such as a red dot or vibration is presented on the cell phone.
Many systems, including HotPaper, Bookmarkr, Mobile Retriever, CAM, described in Parikh, T. S., P. Javid, S. K., K. Ghosh, and K. Toyama, Mobile phones and paper documents: evaluating a new approach for capturing microfinance data in rural India, Proceedings of CHI'06, pp. 551-560, have been proposed for using a camera phone to capture and recognize images of paper documents, and then access the linked digital media. For example, Rohs proposed a system, described in Rohs, M., Real-world interaction with camera-phones, Proceedings of UCS. IPSJ Press (2004), for augmenting real time images of a paper map with dynamic weather information for specific geographical areas. The aforesaid HotPaper enables users, with a camera phone as a Magic Lens, to add and play multimedia annotations linked to text patches in paper documents. The aforesaid Bookmarkr facilitates retrieval and sharing of digital photos with snapshots of printed photos.
Marker-based systems like the one proposed by Rohs as well as CAM depend on 2D barcodes to identify and manipulate the document contents. On the other hand, WikEye, described in Hecht, B., M. Rohs, J. Schöning, and A. Krüger, Wikeye—Using Magic Lenses to Explore Georeferenced Wikipedia Content, Proceedings of the 3rd International Workshop on Pervasive Mobile Interaction Devices (PERMID), pp. 6-10, adopts black dots arranged in a grid as markers. Although easy to detect, the barcodes and dots are often visually obtrusive, require changing the format of original documents, and occupy extra display space on paper. Recent Designable Marker, described in Reilly, D., M. Rodgers, R. Argue, et al., Marked-up maps: combining paper maps and electronic information resources, Personal and Ubiquitous Computing, 2006. 10(4): p. 215-226, improves 2D barcodes by incorporating human-readable components into the mark design, but does not address the issues of document-altering and wasted paper space. To overcome these problems, Data Glyphs and Anoto patterns, well known to persons of skill in the art, were proposed. However, these encoding patterns have to be printed by high resolution printers and captured by specialized cameras for decoding. Moreover, Fujitsu proposed a steganographic printing technique that embeds invisible bar codes into printed color images. All of these visible/invisible barcodes, Data Glyphs and Anoto patterns do not reveal to the human meaningful information about the available media.
To avoid some limitations of marker-based systems, PBAR, described in Hull, J. J., B. Erol, J. Graham, Q. Ke, H. Kishi, J. Moraleda, and D. G. V. Olst, Paper-based Augmented Reality, Proceedings of Int. Conf. on Artificial Reality and Telexistence, pp. 205-209, HotPaper, Mobile Retriever and LLAH, described in Nakia, T., K. Kise, and M. Iwamura, Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval, Proceedings of 7th Int'l Workshop DAS'06, pp. 541-552, use text features, e.g. the spatial layout of words in a text chunk, to identify document patch images. Bookmarkr, MapSnapper, and FIT, described in Liu, Q., H. Yano, D. Kimber, C. Liao, and L. Wilcox, High Accuracy and Language Independent Document Retrieval With A Fast Invariant Transform, Proceedings of ICME'09, pp 386-389, adopt pixel level image features to recognize generic contents including text, pictures and graphic elements. With these systems, visually obtrusive barcodes are not required for identification, the original document formats are intact, and document publishers do not need extra space for barcode printing. On the other hand, since these systems completely remove visible markers on document pages, there is no indication to the user that multimedia information is available at certain locations in the document. Moreover, none of these approaches have marks that can guide users' captures for document patch identification.
Accordingly, new non-obstructive embedded media markers, which can be generated using not only high resolution printing devices but also low-resolution printing devices and methods for implementing and using those markers are needed.